{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieval\n",
    "Many LLM applications require user-specific data that is not part of the model's training set. The primary way of accomplishing this is through `Retrieval Augmented Generation` (RAG). In this process, external data is retrieved and then passed to the LLM when doing the generation step.\n",
    "\n",
    "许多 LLM 应用程序需要用户特定的数据，这些数据不属于模型的训练集。实现此目的的主要方法是通过检索增强生成 （RAG）。在此过程中，将检索外部数据，然后在执行生成步骤时将其传递给 LLM。\n",
    "\n",
    "LangChain provides all the building blocks for RAG applications - from simple to complex. This section of the documentation covers everything related to the retrieval step - e.g. the fetching of the data. Although this sounds simple, it can be subtly complex. This encompasses several key modules.\n",
    "\n",
    "LangChain为RAG应用程序提供了所有构建块 - 从简单到复杂。文档的这一部分涵盖了与检索步骤相关的所有内容 - 例如，获取数据。虽然这听起来很简单，但可能微妙地复杂。这包括几个关键模块。\n",
    "\n",
    "<img src=\"https://python.langchain.com/v0.1/assets/images/data_connection-95ff2033a8faa5f3ba41376c0f6dd32a.jpg\" width=\"70%\" height=\"50%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## Document loaders\n",
    "Document loaders load documents from many different sources. LangChain provides over 100 different document loaders as well as integrations with other major providers in the space, like AirByte and Unstructured. LangChain provides integrations to load all types of documents (HTML, PDF, code) from all types of locations (private S3 buckets, public websites).<br>\n",
    "文档加载器从许多不同的来源加载文档。LangChain提供了100多种不同的文档加载器，以及与该领域其他主要提供商的集成，如AirByte和Unstructured.LangChain提供了从所有类型的位置（私有S3桶，公共网站）加载所有类型的文档（HTML，PDF，代码）的集成。\n",
    "\n",
    "Document loaders provide a \"load\" method for loading data as documents from a configured source. They optionally implement a \"lazy load\" as well for lazily loading data into memory.<br>\n",
    "文档加载器提供了一种“加载”方法，用于从配置的源将数据加载为文档。它们还可以选择性地实现“延迟加载”，以便将数据延迟加载到内存中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='langchain==0.1.20\\nlangchainhub==0.1.14\\nlangchain-experimental==0.0.56\\nlangchain-openai==0.0.8\\nlangchain_anthropic==0.1.13\\nlangchain_chroma==0.1.2\\nfaiss-cpu\\nnltk\\nwikipedia', metadata={'source': '../requirements_0_1.txt'})]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "\n",
    "loader = TextLoader(\"../requirements_0_1.txt\")\n",
    "loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Text Splitting\n",
    "A key part of retrieval is fetching only the relevant parts of documents. This involves several transformation steps to prepare the documents for retrieval. One of the primary ones here is splitting (or chunking) a large document into smaller chunks. LangChain provides several transformation algorithms for doing this, as well as logic optimized for specific document types (code, markdown, etc).\n",
    "\n",
    "检索的一个关键部分是仅获取文档的相关部分。这涉及几个转换步骤，以准备要检索的文档。这里的主要方法之一是将大文档拆分（或分块）为较小的块。LangChain提供了几种转换算法来做到这一点，以及针对特定文档类型（代码、markdown等）优化的逻辑。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text embedding models\n",
    "Another key part of retrieval is creating embeddings for documents. Embeddings capture the semantic meaning of the text, allowing you to quickly and efficiently find other pieces of a text that are similar. LangChain provides integrations with over 25 different embedding providers and methods, from open-source to proprietary API, allowing you to choose the one best suited for your needs. LangChain provides a standard interface, allowing you to easily swap between models.\n",
    "\n",
    "检索的另一个关键部分是为文档创建嵌入。嵌入捕获文本的语义含义，使您能够快速有效地找到文本中其他相似的部分。LangChain提供与超过25种不同的嵌入提供商和方法的集成，从开源到专有API，让您可以选择最适合您需求的一种。LangChain提供了一个标准的接口，让你可以轻松地在模型之间切换。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector stores\n",
    "With the rise of embeddings, there has emerged a need for databases to support efficient storage and searching of these embeddings. LangChain provides integrations with over 50 different vectorstores, from open-source local ones to cloud-hosted proprietary ones, allowing you to choose the one best suited for your needs. LangChain exposes a standard interface, allowing you to easily swap between vector stores.\n",
    "\n",
    "随着嵌入的兴起，出现了对数据库的需求，以支持这些嵌入的有效存储和搜索。LangChain提供与50多种不同矢量库的集成，从开源的本地向量库到云托管的专有向量库，让您可以选择最适合您需求的一种。LangChain公开了一个标准的接口，让你可以轻松地在向量商店之间切换。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrievers\n",
    "Once the data is in the database, you still need to retrieve it. LangChain supports many different retrieval algorithms and is one of the places where we add the most value. LangChain supports basic methods that are easy to get started - namely simple semantic search. However, we have also added a collection of algorithms on top of this to increase performance. These include:\n",
    "\n",
    "一旦数据在数据库中，您仍然需要检索它。LangChain支持许多不同的检索算法，是我们增加最大价值的地方之一。LangChain支持一些简单易上手的基本方法——即简单的语义搜索。但是，我们还在此基础上添加了一组算法以提高性能。这些包括：\n",
    "\n",
    "- Parent Document Retriever: This allows you to create multiple embeddings per parent document, allowing you to look up smaller chunks but return larger context.<br>\n",
    "父文档检索器：这允许您为每个父文档创建多个嵌入，从而允许您查找较小的块，但返回较大的上下文。\n",
    "- Self Query Retriever: User questions often contain a reference to something that isn't just semantic but rather expresses some logic that can best be represented as a metadata filter. Self-query allows you to parse out the semantic part of a query from other metadata filters present in the query.<br>\n",
    "Self Query Retriever：用户问题通常包含对某些内容的引用，该引用不仅仅是语义上的，而是表达了一些逻辑，这些逻辑可以最好地表示为元数据过滤器。通过自查询，可以从查询中存在的其他元数据筛选器中解析出查询的语义部分。\n",
    "- Ensemble Retriever: Sometimes you may want to retrieve documents from multiple different sources, or using multiple different algorithms. The ensemble retriever allows you to easily do this.<br>\n",
    "Ensemble Retriever：有时您可能希望从多个不同的来源或使用多种不同的算法检索文档。Ensemble Retriever 使您可以轻松执行此操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing\n",
    "The LangChain Indexing API syncs your data from any source into a vector store, helping you:\n",
    "\n",
    "LangChain Indexing API将您的数据从任何来源同步到向量存储中，帮助您：\n",
    "\n",
    "- Avoid writing duplicated content into the vector store  <br>避免将重复的内容写入矢量存储中\n",
    "- Avoid re-writing unchanged content <br>避免重写未更改的内容\n",
    "- Avoid re-computing embeddings over unchanged content <br>避免在未更改的内容上重新计算嵌入\n",
    "\n",
    "All of which should save you time and money, as well as improve your vector search results.<br>\n",
    "所有这些都可以节省您的时间和金钱，并改善您的矢量搜索结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain0_1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
